TRl BiS. COPYi 



ESD-TR-71-236 



ESD ACCESSION LIST 

TRl Call No. /yyC2 Y 

Copy No. y of 



ESD RECORD COPY 

RETURfs' , 
,, SCIENTIFIC & TECHNICAL INhOi<,\1ATION DIVISION 

cys. (TRIivifc^d^si210 



THE AIR FORCE JOVIAL COMPILER VALIDATION SYSTEM (JCVS) 



F. Engel, Jr. 



AUGUST 1971 



Prepared for 



DEPUTY FOR COMMAND AND MANAGEMENT SYSTEMS 

ELECTRONIC SYSTEMS DIVISION 

AIR EORCE SYSTEMS COMMAND 

UNITED STATES AIR EORCE 

L. G. Hanscom Field, Bedford. Massachusetts 




Approved for public release; 
distribution unlimited. 



Project 8510 

Prepared by 

THE MITRE CORPORATION 
Bedford, Massachusetts 

Contract F19(628)-71-C-0002 



^-bol^i^'^^ 



When U.S. Government drawings, specifications, 
or other data are used for any purpose other than 
a definitely related government procurement 
operation, the government thereby incurs no re- 
sponsibility nor any obligation whatsoever; and 
the fact that the government may have formu- 
lated, furnished, or in any way supplied the said 
drawings, specifications, or other data is not to be 
regarded by implication or otherwise, as in any 
manner licensing the holder or any other person 
or corporation, or conveying any rights or per- 
mission to manufacture, use, or sell any patented 
invention that may in any way be related thereto. 



Do not return this copy. Retain or destrti 



ESD-TR-71-236 



MTR-2091 



THE AIR FORCE JOVIAL COMPILER VALIDATION SYSTEM (JCVS) 



F. Engel, Jr. 



AUGUST 1971 



Prepared lor 



DEPUTY FOR COMMAND AND MANAGEMENT SYSTEMS 

ELECTRONIC SYSTEMS DIVISION 

AIR FORCE SYSTEMS COMMAND 

UNITED STATHIS AIR FORCE 

L. G. Hanscom Field, Bedford, Massachusetts 




Approved for public release; 
distribution unlimited. 



Project 8510 

Prepared by 

THE MITRE CORPORATION 
Bedford, Massachusetts 

Contract F19(628)-71-C-0002 



FOREWORD 



This report has been prepared by The MITRE Corporation under 
Project 8510 of Contract F19(628)-71-C-0002, The contract is sponsored 
by the Electronic Systems Division, Air Force Systems Command, L. G. 
Hanscom Field, Bedford, Massachusetts. 



REVIEW AND APPROVAL 
This technical report has been reviewed and is approved. 



J^ — '^ 

ROBERT F. JENSEN, Colonel, USAF 

Director of ADPE Selection 

Deputy for Command and Management Systems 



ii 



ABSTRACT 



The Air Force JOVIAL Compiler Validation System (JCVS) was 
developed to assist in the validation of performance of proposed 
JOVIAL language processors. This report discusses the JCVS in the 
context of its use by the Air Force Directorate of ADPE Selection. 
It provides a certification of the audit test modules, and makes 
recommendations for improvements to the JOVIAL audit capability. 
It is recommended that the audit programs be used independently of 
the JCV System. 
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SECTION I 
INTRODUCTION 



The Air Force procedure for the competitive procurement of 
general purpose automatic data processing equipment requires the 
validation of the performance claims made for each proposed ADP system 
in order to establish the responsiveness to the systems requirements 
of the Request for Proposal, Among these requirements, there fre- 
quently occurs the need for a JOVIAL language processor which must 
conform to the AFM 100-24 JOVIAL (J3) Language Specifications. ^^) 
The Air force JOVIAL Compiler Validation System (JCVS) (2) ^^g developed 
under contract to assist in the task of validation of the performance 
of proposed JOVIAL language processors. This report briefly discusses 
the JCVS in the context of its possiole use by the Air Force Direc- 
torate of ADPE Selection, MCS , suimnarizes the experience acquired in 
working with the JCVS, critiques the audit test modules, and makes 
recommendations for improvements to the JOVIAL audit capability. 

The compiler validation process or audit function has two 
purposes: the first is to establish that each language feature of 
the Air Force standard programming language is accepted by the JOVIAL 
processor under examination; the second is to establish that the 
execution of each language feature produces the prescribed results. 
The JCVS accomplishes this audit function by presenting to the JOVIAL 
processor a set of one or more JOVIAL source programs and their 
required input data (if any) , which together contain statements 
invoking each of the required standard JOVIAL features. Such programs 
are referred to as "Audit Programs." Upon successful compilation 
and execution of the audit programs, it is established that a processor 
does conform to the standard for those features tested. While it is 
impossible for the audit programs to check for the correct implemen- 
tation of all possible combinations of standard features due to the 
large number of tests that would be required, it is expected that a 
sufficiently representative sample will be included to establish 
reasonable confidence in the capability of the observed system. 



SECTION II 
THE JCVS COMPONENTS 



The heart of the Air Force JCVS is the Population File. It 
contains all of the JOVIAL source statements comprising the tests of 
the standard features and the auxiliary procedures for reporting the 
results of execution of the tests. In addition to the Population 
File, the JCVS consists of a Population File Maintenance program, a 
Selector program, a Source Program Maintenance program, and a con- 
version program for character set transformation. These auxiliary 
programs are written in ANS COBOL and provide the means for automating 
the manipulation of the constituent elements of the JOVIAL audit 
programs . 

THE POPULATION FILE 

The Population File (Pop File) is the data base for the JCVS. 
It contains approximately 9000 cards of JOVIAL source statements 
and commentary which are subdivided into test modules of, at most, 50 
cards each. Each test module contains all of the JOVIAL statements 
necessary to effect the test of a given feature including all item 
declarations, procedure declarations, etc. The test module is 
completely self-contained, with the exception of the output procedures, 
which are invoked to record the results of the test. The report 
writing or output procedures are contained in a separate module, 
providing a uniform reporting mechanism. Thus, each test module 
as it exists in the Pop File may be compiled and executed independ- 
ently of every other test module. 

The test modules may also be compiled and executed together as 
a single JOVIAL program by combining the individual modules in any 
desired sequence. Through the use of appropriate conventions in 
assigning names to variables, labels, procedures, etc., duplication 
of names has been avoided. The characteristics of the J3 language 
are such that no rearrangement or sorting of the JOVIAL statements 
of the test modules is necessary.* 



*In this regard the J3 language differs significantly from both 
FORTRAN and COBOL. In FORTRAN specification statements must precede 
all executable statements, and in COBOL appropriate statements must 
appear in the ENVIRONMENT, DATA and PROCEDURE divisions of the program 
in that order (cf MTR-1953,The Air Force COBOL Compiler Validation 
System (CCVS))^ 



The physical arrangement of the Pop File consists of 4000 
characters, fixed length records, each containing the characters fron 
fifty 80-character cards. The first card of each record is a header 
card which identifies the module and may also designate a dependence 
upon other specific modules. The header card is in the form of a 
JOVIAL comment line and is considered to be a part of the JOVIAL 
test module. If more than 50 cards are required for a test, there 
is provision on the header card to specify that the module is to 
be appended to the preceding module as a continuation of it. Finally, 
the header card may designate that the module is to be present in 
every audit program, whether or not its selection has been requested. 
Characters 73-76 of every card contain the module identification num- 
ber, and characters 78-80 contain the card sequence number within 
the module, which must be in the range of 1 to 50 inclusive. The 
77th character of the header card designates which of the foregoing 
functions is intended for that module. On all other cards, this 
character designates JOVIAL source statements. The module identifi- 
cation number, the card sequence number and the function character 
provide the key basis for the functional manipulations performed 
upon the Pop File by the other components of the JCV System. 

The first record of the Pop File has the module identification 
number 001, and fulfills additional unique functions. The first five 
cards of that record may be used to convey to the user for informa- 
tional purposes only data concerning the source of the particular 
Pop File and the hardware environment in which it may operate. The 
content of these cards is ignored by the JCVS . Succeeding cards in 
the first record are designated environmental software cards, and 
may be operating system control cards, JOVIAL processor control cards, 
etc., which are to be placed before and/or after every audit program 
when it is selected from the Pop File. By offsetting these environ- 
mental software cards by one character, the JCVS is able to handle 
operating system control cards which require special characters in 
Column 1. 



THE POPULATION FILE MAINTENANCE PROGRAM 

The Population File Maintenance Program (POPFM) provides a 
mechanism for creating a new Pop File either from cards for the 
individual modules, or by updating an old Pop File by means of dele- 
tion, replacement or insertion of either individual cards or entire 
modules on the basis of the module identification numbers and card 
sequence numbers. A report of the functions performed is generated 
by POPFM. . 



THE SELECTOR PROGRAM 

The JCVS Selector Program (SELECT) operates on the Pop File to 
produce a single JOVIAL source program from one or more JOVIAL test 
modules residing on the Pop File. The user designates, by means of 
control cards, those modules which are to be selected. The selected 
modules appear in order by module number as they exist on the Pop 
File. The user may direct the selectrd modules to be written onto 
magnetic tape or punched cards. An audit file is produced by SELECT 
containing a record of the modules selected and additional messages 
relating to suspected error conditions encountered. 



THE SOURCE PROGRAM MAINTENANCE PROGRAM 

The JCVS Source Program Maintenance Program (SOPMM) operates on 
an existing JCVS Source Program File to update and generate a new 
Source Program File. The SOPMM provides the user with the ability to 
add information to the source program file, delete information from 
the source program file, or replace information on the source program 
file on a card image by card image basis, according to the module 
identification number and the card sequence number. The user may 
direct the new source program file either to magnetic tape or punched 
cards. An audit file is produced optionally by SOPMM providing a 
record of the diagnostic and trace messages as well as a source 
program listing when desired. 



THE POPULATION FILE INITIATING PROGRAM 

The JCVS Pop File Initiating Program (INIPOP) operates to 
initiate and, at the user's option, to re-number a Pop File either 
from an existing Pop File or from a card file containing the test 
modules. The module identification numbers only are re-sequenced as 
directed by user supplied control cards. Certain cross-references 
between modules are changed automatically when affected by the re- 
numbering process. An audit report is created which contains 
diagnostic messages, and an optional listing of JOVIAL source state- 
ments on the new Pop File. A punched card deck of the Pop File may 
be obtained as an optional output. 



THE REPORT WRITER PROGRAM 

The JCVS Report Writer P rogram (JCVSRP) operates on the Pop 
File to produce a listing of the test modules on the Pop File or 
a listing of all the test header cards on the Pop File, or both as 
directed by user supplied control cards. 

The reader is referred to the JCVS User's Manual ^ for a more 
detailed description of the JCVS components and their use. In the 
work with the JCVS Test Modules which was performed on both the 
GE-635 at RADC and the IBM/360-50 at MITRE, it was found more con- 
venient to manipulate the test module card decks manually, rather 
than to use the JCVS programs enumerated above. The JCVS programs 
require that control cards be prepared in addition to the JOVIAL 
statement modifications, their use requires additional computer runs 
to be made to generate the modified audit programs before they can 
be processed by the JOVIAL compiler, and the reruns occasioned by 
errors introduced in these peripheral activities lengthen the 
apparent turn around time to process an audit program with the JOVIAL 
compiler. However, the JCVS components have each been used at least 
once, and they appear to function as described. It was found 
necessary to introduce a few minor modifications to initialize print 
records, and repair clerical errors. 



SECTION III 
THE JCVS POPULATION FILE 



The present set of JCVS Test Modules contained in the Population 
File fall into four general categories: 

1. The Pass/Fail Reporter and its verification tests 

2. Tests of Declarations 

3. Tests of Procedural Statements 

4. Tests of Processing Declarations 

Each of these categories of tests is discussed in this section 
with respect to its operational characteristics, the thorouhgness 
and rigor of test logic, and the completeness of coverage of the 
AFM 100-24 JOVIAL features. 



PASS /FAIL REPORTER 

The Pass/Fail Reporter consists of three JOVIAL Procedures 
contained in Test Module 9998. Their function is to print one line 
identifying the test module and a second line indicating that the 
test was successful or had failed. The Procedure OUTERR prints a 
line by calling a FORTRAN subroutine in which the information to be 
printed is the 40-character Hollerith argument. This FORTRAN sub- 
routine is not provided, nor are any specifications for it given. It 
is not obvious how a standard conforming FORTRAN program could be 
made to manipulate successfully this Hollerith argument. 

The procedure OUTERA is invoked at the beginning of execution 
of each test to set up the test module identification line. The 
4-character Hollerith argument of this procedure is the identifi- 
cation number of the test, which is then combined with the invariant 
part of the message by use of the JOVIAL OVERLAY feature. The 
message is printed by invoking the procedure OUTERR. 

At the conclusion of its execution, each test module invokes 
the procedure OUTERB to set up and print the Pass/Fail message. The 
argument of the procedure OUTERB is a 1-character Hollerith variable 
which is assumed to have been set to the value "Y'* if the test were 
successful, and to "N" if the test failed. The EQ relational opera- 
tor is used in an IF: clause to compare the argument with the Hollerith 
literal constant 1H(Y) . If the condition is true, the 40-character 
Hollerith variable is assigned the value of the "test successful" 
message, and OUTERR is invoked to print it. If the condition is not 



true, then the variable is assigned the value of the "test failed" 
message and the procedure OUTERR is invoked by the same statement. 

The Pass/Fail Reporter provides a uniform reporting mechanism 
for all tests, and localizes all output statements and interfaces 
with FORTRAN in this one module. This particular embodiment offers 
some disadvantages. The use of other than standard JOVIAL output 
procedures makes the Audit Programs processor dependent, so that they 
must be specifically adapted to each processor to be tested. The 
FORTRAN interface with a Hollerith argument introduces a dependency 
upon the processor under test. The Pass/Fail Reporter is complicated 
by the use of nested procedures, i.e. procedures invoking other pro- 
cedures; by the use of argument passing between procedures; by the 
use of logic within the Reporter; and by the use of the OVERLAY 
feature. Thus, the Pass/Fail Reporter is more sophisticated and com- 
plex in terms of JOVIAL features employed than many of the tests 
which use it to report their success or failure, and might itself be 
a source of error, preventing the use of any of the audit programs. 
The use of the OVERLAY feature might introduce processor dependencies 
due to word boundaries in data which are to be written out. Finally, 
the Pass/Fail Reporter provides no information on the test results 
which can be used either to confirm the success or to assist in 
diagnosing the failure of a test. 

The function of the five test modules 0500-0520 is to validate 
the Pass/Fail Reporter. As originally designed, these modules 
attempted to verify the correct functioning of a processor for each 
of the JOVIAL language features which are employed in the Reporter. 
To do this independently of the Reporter itself, non-standard JOVIAL 
output statements were employed to print the test results. These 
statements had been subsequently replaced with calls to the Pass/ 
Fail Reporter procedures for printing, with the result that the 
desired independent verification of these features has not been 
achieved. A further defect in these tests is that the features 
presumed to be tested are not tested under the same conditions of 
use in the Pass/Fail Reporter. For example, the Pass/Fail Reporter 
uses an IF: clause having a Hollerith: literal: relation list as its 
boolean: formula , but in test nodule 0520 the IFrclause feature is 
tested using only the boolean: constants and 1 for the boolean: for- 
mula . Certainly different machine code is required for the relational 
comparison of literal formulas than for boolean: constants , and the 
success of the test of the latter case does not imply that the pro- 
cessor will treat the former correctly. Further, a sophisticated 
optimizing processor mi^ht recognize the invariance of the boolean: 
constants and generate code which would ignore completely the condi- 
tional statement as employed in test module 0520, thus completely 
invalidating the purpose of the test. 



The following JOVIAL features are those presumed to be tested 
by modules 0500-0520: 

1. Hollerith Item Description 

2. Preset Hollerith Item 

3. Procedure Call using Hollerith Argument 

4. Hollerith Assignment Statement 

5. GOTO Statement 

6. IF:clause , boolean: constant 

7. Procedure Definition, with no Argument 

8. Procedure Call, with no Argument 

The Pass/Fail Reporter Module 9998 uses all of these features 
except the last two items, and in addition, employs the OVERLAY 
feature, the definition of a procedure having a Hollerith argument, 
and the nesting of procedures with argument passing required. These 
last features are not explicitly validated by test modules 0500-0520. 

A few specific programming errors were noted in module 9998. 
The global variables OUTl, OUTA, OUTB were not defined. Since these 
variables are referenced in almost every module, it is necessary 
that their item: descriptions occur at the beginning of the audit 
programs. The argument AA of the procedure OUTERR(AA) was not passed 
to the output subroutine, so that the message to be printed was lost. 
Finally, in the definition of the procedure OUTERA(CC), its argument 
was illegally declared to OVERLAY another datum. Since at execution 
time the dummy argument CC is replaced by any number of actual 
arguments, each of which may be defined in a different calling pro- 
gram unit, such use of OVERLAY is inconsistent. 

In view of the difficulties enumerated above, MITRE provided 
a new module 0198 to replace test module 9998 as the Pass/Fail 
Reporter. This module is listed in Appendix I. The procedure names 
and argximents have been retained to be compatible with the existing 
procedure calls, but each procedure is now independent of the others; 
i.e. there is no nesting of proceaures and concommitant argument 
passing. Global variables are used to transmit information to the 
procedures rather than relying on the procedure arguments. The use 
of the OVERLAY feature was retained, but it now occurs at the global 
level, rather than within the output procedures, and it does not 
Involve the procedure arguments. This could be avoided by using a 
larger variety of print files to accommodate the different message 
lengths, by making all data for output the same length, (e.g. OUTA, 
OUTB and OUTl), or by introducing a more complex feature such as an 
array or table for constructing the output message. Short of 
revising the entire mechanism, it was felt that this represents the 
best compromise. A further modification was made to cause the 
value of the Pass/Fail indicator to be printed as part of the Pass/ 
Fail Message, and to reset this parameter co the failed condition, 
i.e. 0UTA=1H(N) , after each success or fail message is printed. As 



a result the next test module must set the parameter to "Y", other- 
wise the default condition would be that the test failed. 

Also provided is a new test module 0100 to validate chis Pass/ 
Fail Reporter. Rather than attempt to validate independently each 
of the features employed therein, it assumes that all of the feature's 
are implemented correctly, and attempts to demonstrate that each of 
the three procedures does indeed produce the expected results when 
called with appropriate parameter values. One line is printed 
directly by a JOVIAL OUTPUT statement in this test module when the 
print file is opened. A second line is printed by a call to the 
procedure OUTERR, and a third line results from a call to the pro- 
cedure OUTERA, which incorporates the four-character module number 
into the test header line. Then to test the Pass/Fail function^ the 
procedure OUTERB is called, first with the parameter OUTA set to 
"N", then again after setting OUTA to "Y" to demonstrate that the 
appropriate pass and fail messages are printed in each instance. 
Then, OUTERB is called a third time to demonstrate that the para- 
meter OUTA had been reset to "N" after printing the "test succeGsfuP* 
message. The test module then prints a final line using a direct 
JOVIAL OUTPUT statement, indicating the end of the test and the 
expected number of lines which are printed. The auditor must e;:aminc 
the output and determine that the appropriate messages have been 
printed in the expected sequence. 

The JOVIAL file: declaration which defines the print file is 
included in the test module 0100, as are also the item: descriptions 
for the global parameters used by the audit programs. It is there- 
fore necessary that modules 0100 and 0198 must be present in every 
audit program, and that they should be positioned at the beginning 
of the program. Hence, the relatively low module numbers which have 
been given them. The AFM 100-24 provides that the device: name 
occurring in the file: declaration is to be defined by each implcmcn- 
tor. It is therefore to be expected that this statement which occurs 
only once in module 0100 will have to be changed for each processor 
to be tested. 

In testing the JCVS with the J3 processor on the IBM Gysteni/360 
Model 50, the efficacy of the revised Pass/Fail Reporter and valida- 
tion procedure was demonstrated, when it was found that the "test 
successful" message was always printed, regardless of whether the 
parameter OUTA had the value "Y" or "N". This was due to a pro- 
cessor deficiency, which would not have been discovered by the 
original verification procedure. 



DECLARATIONS 

There are 151 test modules which are designed to test the 
declaration features of the JOVIAL language. Each test references 
the appropriate section of AFM 100-24 which describes the feature 
being tested, and the set of data declarations for simple items, 
arrays and tables of all types appears to be reasonably complete 
vuth respect to the variety of permissible forms which are represented. 

The item declarations are the means for associating names with 
elements of a program and for describing the attributes of such 
elements. Item declarations are usually non-executable; i.e. they 
do not of themselves result in the generation of executable machine 
instructions. They provide information to the processor useful at 
compile time only, affecting storage allocation and the form of 
executable machine instructions chat are generated for other state- 
ments. For this reason, the set of JCVS tests which are designed to 
validate the item definition features, e.g. modules 1000-1235, are 
inconclusive in themselves as to what features are or are not 
implemented correctly. Each of these tests upon execution will 
always produce the "test successful*' message, signifying only that 
the test module was included in the audit program. The message 
has no significance relative to the capability of the JOVIAL pro- 
cessor under test to accept and interpret the feature. Careful 
analysis is required on the part of the auditor to validate the 
capability of the processor to handle these features. The compile 
time processor output might be helpful to the auditor, but it, too, 
may be inconclusive. Error flags or diagnostic information, if 
present, indicate things the processor does not recognize as valid, 
but the absence of such commentary does not mean that the features 
have been correctly interpreted and implemented. In fact a poor 
diagnostic or error detection facility in the processor would pro- 
duce the same results. 

As a further specific example, tnere occurs in test modules 
1200 and 1210 the use of the range declaration for integer and fixed 
type data. According to the language specifications, the processor 
makes use of this information to provide adequate scaling and pre- 
cision of representation of intermediate results. In order to test 
this feature, it would be necessary to define data and formulas using 
them which would demonstrate that scaling of the intermediate results 
was correct. This has not been done in these or any other of the 
tests included in the Pop File, 

In summary, these tests give no assurance that the processor 
has interpreted each feature correctly, since this can only be done 
by the execution of appropriate statements which reference the items 
and validate the results. 

10 



PROCEDURAL STATEMENTS 

There are 162 test modules in the Pop File covering the features 
of the procedural statements in the JOVIAL language. These modules 
also reference the appropriate sections of AFM 100-24 which describe 
the features being tested, and the systematic approach which has been 
followed seems to have provided for the inclusion of all of the 
essential features in the tests, with the exception that tests of all 
Input/Output features have been purposely excluded, as have tests of 
the DIRECT: statement (i.e. in-line machine code). These are con- 
sidered to be processor dependent features (cf.(2) pp. 1, 5). 

In general these tests are simple and straightfbn:ard in 
design and adequately validate the acceptance of the features being 
tested. The inaccessability of the computed results and test values 
complicate the analysis of test failures. Some of the test modules 
make use of features other than those under test, so that a test 
failure might be attributed to the wrong cause, again complicating 
the auditor's analysis of the failure. While the JCVS design made 
provision for indicating interdependencies among the test modules 
on the test header card, we found no instance of its use to ensure 
that dependent features would be tested automatically. 

The tests of arrays and tables generally use a snail number of 
entries; most often 2 in the case of tables, and sometimes 1 for an 
array dimension. While these cases are consistent v.^Lth the rtcindai-d, 
a more comprehensive test of larger structures would be desirable to 
demonstrate the capability of the processor to handle such references. 
It is also to be noted that many of the tests (ci module 3fi55) rely 
on the successful referencing of a single element of an array or 
table for validation. Again, a more comprehensive testing of many 
element references would provide better assur'.icn Ol the processor's 
capability. 

A detailed analysis of one of the more complex test modules, 
module 5310, suggests several ways in which the tests might be 
improved from the standpoint of the audit function. Module 5310 
tests the alternative: statement ; i.e., the use of IFEITH. . .ORIF. . . 
The JOVIAL alternative: statement provides for the sequential testing 
of any number of conditions, each of which except Lhe first occurs 
in a successive or: if: clause . Associated with each condition is a 
single independent statement which will be executed if ^^nd only if 
that condition is true and every condition preceding it is false. 
Following execution of the associated statement, c^-ntrol is to be 
returned to the point which would follow the last or: if: clause if 
none of the set of alternative conditions were t"ae. Thus, for an 
n-condition alternative there are in general n+1 paths leading to the 
following statement. 

11 



In the test module 5310^ there are eight alternative: statements , 
two of them having three alternative conditions, the others having 
just two alternative conditions in each. The expected normal execu- 
tion of the program causes each alternative statement to be executed 
only once, so that the remaining two or three possible alternative 
paths are not tested for each statement. Instead, this test module 
demonstrates different types of branching; i. e. sequences of alter- 
native conditions in different alternative: statements , which are 
themselves differently structured. Hence, it does not conclusively 
demonstrate that the processor does indeed correctly handle the set 
of alternatives. Several of the conditions in these alternative: 
statements are in the form of the boolean constants or 1; i.e., 
they are invariant. A sophisticated JOVIAL processor with optimiza- 
tion might recognize these and generate different machine code, by- 
passing the tests that would be required for the general, variable 
alternatives. Thus, again, this method of testing does not provide 
assurance that the processor has correctly implemented or interpreted 
the feature. Finally, the logic of the test provides multiple paths 
to the successful conclusion of the test, so that a combination of 
errors in the interpretation of the alternative: statements could 
cause the test to appear to be successful. The points just enumerated 
reflect the primary emphasis of the JCVS tests on the demonstration 
of the capability of a processor to accept the standard features, and 
the lesser concern with tne thorough demonstration of the correctness 
of interpretation or implementation of the features. 

Modules 5311 and 5312 have been developed to illustrate the more 
thorough testing of the alternative: statement . In module 5311, a 
five condition alternative: statement is to be executed six times, 
exercising each possible branch. Upon completion a check is made to 
verify that each path had been traversed as expected. In module 5312, 
the same five conditions are employed, but internal statement labels 
and branching to them are added to demonstrate this additional capa- 
bility required within an alternative: statement . A call to a pro- 
cedure is also introduced to be executed as one of the branches. 
Listings of these two modules are included in the recommended changes 
in Appendix I. 



PROCESSING DECLARATIONS 

There are 43 test modules devoted to the JOVIAL processing: 
declarations . These features are switches and user defined closes, 
procedures and functions. Here, too, each test module references 
the appropriate section of AFM 100-24 describing the feature being 
tested. This set of tests seems adequate in terms of the variety of 
forms and features in this category which are included in the tests. 
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SECTION IV 
VALIDATION OF THE JCVS TEST MODULES 



In order to establish validity of the JCVS test modules for 
performing the audit functions, they were reviewed to ensure that: 

1. They were free of mechanical and clerical errors; 

2. They were free of syntactic errors, i.e. the JOVIAL 
source statements conformed to the syntactic forms 
of AFM 100-24; 

3. They were free of logic errors; i.e. the results of 
each test were consistent with the requirements of 
AFM 100-24; 

4. They tested a sufficient number and variety of the 
language features to provide an adequate sample of 
the AFM 100-24 requirements. 

The adequacy of coverage of the JOVIAL language features is dis- 
cussed in the previous section under the appropriate categories. The 
cross reference list* of features tested by the set of modules in 
the JCVS Pop File was compared to the features defined in AFM 100-24 
on a paragraph-by-paragraph basis. No glaring omissions were dis- 
covered, and on a purely subjective basis it was concluded that the 
coverage was adequate for the intended purpose, with the reservations 
previously cited concerning the intentional omission of features from 
testing, the insufficiency of testing of alternatives and the lack of 
logical rigor of the tests. The set of tests is open-ended, and it 
is presumed that tests of the input/output features and of more com- 
binations of features will be added in the future. 

A two-phase procedure was followed in carrying out the error 
investigation. First, selected test modules were subjected to a 
desk review and analysis in which the source program listings were 
examined and checked for clerical errors and consistency with 
AFM 100-24. In some few instances flow charts were prepared to 
assist in the validation of the logic of the tests, as for example 
in the case of module 5310 cited above. Secondly, the test modules 
were subjected to machine processing on two different systems to 
provide an independent and more complete verification of the syntac- 
tic analysis and correctness of execution results for each test. 

(U) 
The GE-600 Line JOVIAL processor^ "^ was selected for testing 

the JCVS modules because of ready access to the system on the GE-635 

at the Rome Air Development Center and because that processor was 



•^^-See reference 2, p. 196-225. 
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thought to conform closely to the AFM 100-24 language, the known 
exceptions being the Input/Output features, and the DUAL and STRING 
item: declarations . The entire JCVS Pop File was divided into arbit- 
rary subsets each containing from 50 to 70 test modules from which 
were removed those modules requiring features known not to be implemen- 
ted in the CE-600 Line JOVIAL processor. The subsetting was done to 
avoid exceeding the limitations imposed by the compiler, to increase 
the probability of successful compilation and execution of at least 
some of the test modules during the brief test period, and to enable 
analyses of the results of some runs to be made while others were in 
the processing queue. A copy of the Pass/Fail Reporter, module 9998, 
modified as described below, was incorporated into each subset to 
make each a separate audit program for independent compilation and 
execution. The set of job control cards shown in Table I was 
evolved through an initial series of eight runs, in which the principal 
difficulties encountered were in the setting of memory limits, the 
proper identification of Input/Output units and files, and in the 
interpretation of system error messages. These control cards were 
attached to each audit program deck to effect the desired processing. 
At least two runs were made with each subset with the exception of 
the subset containing modules 5190 through 6085. That subset was 
run only once, and the run aborted after execution of all but five 
of the test modules in the set. 

After each run, the printed output was examined for error 
diagnostic indications which may have occurred during compilation. 
If the cause of the error could be determined quickly to be in the 
test module and an obvious correction could be made, this was done 
and the module was left in the audit program for re-processing. 
Otherwise, the test module in question was removed from the deck 
before another attempt at processing was made. It was observed 
that the GE-600 Line JOVIAL processor would ignore invalid state- 
ments during the code generation phase of compilation, and the pro- 
gram would execute with those parts missing. This would frequently 
result in the "Test Successful" message being printed for that test 
module when in fact the test had not been performed properly. There- 
fore, ic was necessary to examine carefully each run to verify 
successful execution of the tests. 

As a result of this processing, 86 JCVS test modules were com- 
piled and executed successfully; 178 modules were compiled without 
generating any error indications but were not executed due to prior 
run termination; 46 test modules were found to contain problems 
requiring further analysis to determine the nature of the error and 
whether the processor or the test module was at variance with 
AFM 100-24; and 49 test modules were not processed. Modifications 
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Table I 
GE-635 Job Control Cards for JOVIAL Compile and Execute Run 

$ SNUMB 25460 

$ IDENT EMBIL,RROBINSON,JCVS ,IH519,5581 

$ OPTION ERCNT/4500/ , JOVIAL 

$ JOVIAL NDECK 

$ LIMITS 08, 32000, , 10000 

$ INCODE IBMEL 

** source deck goes here *^'^ 

$ EXECUTE 

$ ENDJOB 
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were made to ten of the test modules to achieve these results. 
These modifications are included in Appendix I, and the results are 
detailed in Table 2, Column 4. 

The modified Pass/Fail Reporter module 9998 used the GE-600 Line 
JOVIAL output statements rather than requiring a FORTRAN subprogram 
for the output function because there was insufficient information 
to define the interface between the GE-600 Line JOVIAL and FORTRAN. 
While the GE-600 Line JOVIAL input/output features differ from those 
of the AFM 100-24 language, they are used here only to perform the 
reporting function, which usage does not interfere with the validity 
of the tests being reported. The specific modifications consisted of 
a file declaration statement and a file opening statement placed at 
the beginning of the audit program, and an output statement to 
replace the procedure call on line 9998J011, as follows: 

FILE PRNT V(NORM) R06 $ "DECLARE PRINT FILE" 

Oai(l,PRNT) $ "OPEN PRINT FILE" 

109998(3, PRNT, 0,AA, 7) $ "PRINT PASS/FAIL, ETC." 

(3) 
The IBM System/360 JOVIAL Compiler was selected as the 

second machine processing test vehic le because it appeared to be 
the most compiece implementation of the JOVIAL language and because 
it became available on the MITRE/360-50 system shortly after the 
work at RADC had demonstrated the usefulness of machine testing of 
the ^udic programs. The procedure followed was similar to that 
employed with the GE-635, except that only chose test modules which 
had not executed successfully with the GE-600 Line JOVIAL were 
processed on the system/360. A new Pass/Fail Reporter module 0198 
previously described was used in place of module 9998 in all /360 
JOVIAL runs. When, after each run, examination of the printed out- 
put established the successful compilation and execution of a test 
module, chat module was considered to be validated and was removed 
from further processing. Those test modules which either failed to 
execute successfully or had errors detected in compilation were 
analyzed further in detail. Corrections were made to sixteen modules 
which were found at variance with AFM 100-24. The corrected modules 
were processed again until they, too, were compiled, executed and 
verified successfully. As a result of this processing, 178 
additional JCVS test modules were compiled and executed successfully. 
There remain 102 test modules whose status has not been resolved; 
i.e. it has not been established conclusively that these test 
modules are in conformance to AFM 100-24 requirements , although 31 
of them did compile without error diagnostics. A complete summary 
of the validation results is presented in Appendix II. 
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The successful compilation and execution of the test modules 
when confirmed by examination of processor generated output, has 
been accepted as validation of the modules because it is unlikely 
that the same error in interpretation will be made in both the test 
module and the processor, and fail to be noted by the reviewer. On 
the other hand, the failure of a test module in either compilation 
or execution is not conclusive evidence that the test module is 
incorrect. It has been established that errors exist in both the 
GE-600 Line JOVIAL and the IBM/360 JOVIAL Processors. While it was 
beyond the scope of the present investigation to analyze processor 
failures and make the necessary corrections, it has been possible 
in a few instances to identify specific processor shortcomings, as 
for example, the IBM processor's failure to reference correctly a 
procedure argument in making a boolean comparison, and also the 
failure to branch correctly on an index switch. In these cases the 
generated code was at fault although the interpretation of the JOVIAL 
source language was correct. Aside from the 39 test modules involv- 
ing dual items, there are 34 test modules using Table items which 
are in an unresolved status. It is suspected, but not verified, 
that the processor is responsible for these failures, and that ;;he 
test modules are correct. 
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SECTION V 
SUMMARY AND RECOMMENDATIONS 



The primary objective of this study of the JOVIAL Compiler 
Validation System was to establish the validity of the set of tests 
with respect to their correctness and consistency with the AFM 100-24 
Language Specification. Desk analysis and machine processing with 
the GE-600 Line JOVIAL and the IBM System/360 J3 JOVIAL verified 
that 260 of the JCVS test modules, including those corrected, meet 
the criteria for use in performance validation by MCS. An additional 
seventy- three test modules are satisfactory on the basis of analysis 
alone, but could not be verified by processing due to the absence of 
correct processor implementations of dual items and table features. 
There are thirty test modules which have not been validated by any 
means. They, too, might be used in the performance validation, with 
the qualification that they may be modified when and if discrepancies 
are established by processor rejection and subsequent analysis. 

The recommended changes and corrections to the test modules to 
achieve these results are documented in Appendix I. They have been 
incorporated into the Population File, and the set of corrected 
test modules on both magnetic tape and punched cards has been pro- 
vided to the Air Force Directorate of ADPE Selection, together with 
a printed listing of the source cards for the test modules. 

The investigation of the JCVS test modules also established that 
there are areas in which improvements can be made with respect to the 
overall audit function. It is recommended that future work be 
undertaken 1) to improve the rigor and thoroughness of the tests, as 
exemplified by the MITRE developed modules 5311 and 5312, and 2) to 
provide the auditor with more useful information about the test 
results to aid in his analysis. These improvements would on the 
one hand provide further assurance of the correctness of processor 
implementation, and on the other enhance the usefulness of the JCVS 
in the MCS performance validation environment. 

In the process of carrying out this study of the audit programs, 
the JCVS auxiliary programs were used on both the IBM/ 360 and 
GE-635 systems to prepare the punched card decks for processing. The 
use of the JCVS auxiliary programs was found to require more time 
and to be less convenient than manual preparation of the card decks. 
It is recommended that MCS distribute the JCVS audit programs on 
tapes directly to the vendors independently of the JCV System. 
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APPENDIX I 
MODIFICATIONS TO THE JCVS TEST MODULES 



The corrections to the JCVS test modules which were necessary 
to make them conform to the AFM 100-2A requirements, to correct 
clerical errors, or to improve the performance with respect to the 
audit function are cited in Table II. The format is that obtained 
by listing the JOVIAL source language cards for the changes. The 
four digits in columns 73-76 are the module numbers, and the last 
three digits give the card sequence numbers. These cards are to 
replace the same numbered cards in the Pop File. Also contained in 
this list are completely new modules 0100, 0198, 5311 and 5312 which 
are discussed in the text. 
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TABLE II 
JCVS Test Module Corrections 
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TABLE II (Continued) 
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TABLE II (Concluded) 
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APPENDIX II 
RESULTS OF THE JCVS TEST MODULE VALIDATION 



In Table III the entire set of JCVS Test Modules are listed by 
module number, indicating the results of machine processing and 
analysis for each module. The first column of this table indicates 
those modules which were modified by MITRE, with an asterisk desig- 
nating those modules which were newly created by MITRE. In the 
next tv;o columns are indicated the results of the machine processing 
on the GE-635 and IBM/360, respectively. The significance of the 
symbols appearing therein is as follows: 

S - Successful compilation anJ execution without error 

C - Compilation without error 

M - Successful compilation and execution without error 

after modification by MITRE 
F - Failed in execution after compilation without error 
N - Errors detected during compilation 
W - Withheld from processing for known deficiency in 

processor, e.g. dual item, 

A mark in the fourth column indicates that by analysis of the 
source program and processing results, if any, the test module com- 
plies with the requirements of AFM 100-24. The last column of the 
table indicates those modules whose status is unresolved; i.e. there 
has been no confirmation of validity by successful machine proces- 
sing of the module. A letter D in this column identifies test 
modules involving Dual Items, for which no processor implementation 
exists. 
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Table III (Continued) 
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Table III (Continued) 
JCVS Test Module Validation Summary 
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Table III (Continued) 
JCVS Test Module Validation Smnmary 
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JCVS Test Module Validation Summary 
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JCVS Test Module Validation Summary 
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Table III (Continued) 
JCVS Test Module Validation Summary 
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Table III (Continued) 
JCVS Test Module Validation Summary 
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Table III (Continued) 
JCVS Test Modules Validation Summary 
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Table III (Continued) 
JCVS Test Module Valiation Summary 
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Table III (Continued) 
JCVS Test Module Validation Summary 
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Table III (Continued) 
JCVS Test Module Validation Summary 
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